<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<title>Chapter 1 Introduction | Data Science at the Command Line, 2e</title>
<meta name="author" content="Jeroen Janssens">
<meta name="description" content="This book is about doing data science at the command line. My aim is to make you a more efficient and productive data scientist by teaching you how to leverage the power of the command line....">
<meta name="generator" content="bookdown 0.24 with bs4_book()">
<meta property="og:title" content="Chapter 1 Introduction | Data Science at the Command Line, 2e">
<meta property="og:type" content="book">
<meta property="og:url" content="https://datascienceatthecommandline.com/chapter-1-introduction.html">
<meta property="og:image" content="https://datascienceatthecommandline.com/og.png">
<meta property="og:description" content="This book is about doing data science at the command line. My aim is to make you a more efficient and productive data scientist by teaching you how to leverage the power of the command line....">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:title" content="Chapter 1 Introduction | Data Science at the Command Line, 2e">
<meta name="twitter:description" content="This book is about doing data science at the command line. My aim is to make you a more efficient and productive data scientist by teaching you how to leverage the power of the command line....">
<meta name="twitter:image" content="https://datascienceatthecommandline.com/twitter.png">
<!-- JS --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.6/clipboard.min.js" integrity="sha256-inc5kl9MA1hkeYUt+EC3BhlIgyp/2jDIyBLS6k3UxPI=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/fuse.js/6.4.6/fuse.js" integrity="sha512-zv6Ywkjyktsohkbp9bb45V6tEMoWhzFzXis+LrMehmJZZSys19Yxf1dopHx7WzIKxr5tK2dVcYmaCk2uqdjF4A==" crossorigin="anonymous"></script><script src="https://kit.fontawesome.com/6ecbd6c532.js" crossorigin="anonymous"></script><script src="libs/header-attrs-2.9/header-attrs.js"></script><script src="libs/jquery-3.6.0/jquery-3.6.0.min.js"></script><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link href="libs/bootstrap-4.6.0/bootstrap.min.css" rel="stylesheet">
<script src="libs/bootstrap-4.6.0/bootstrap.bundle.min.js"></script><link href="libs/_Source%20Sans%20Pro-0.4.0/font.css" rel="stylesheet">
<link href="https://fonts.googleapis.com/css2?family=Fira%20Mono:wght@400;600&amp;display=swap" rel="stylesheet">
<script src="libs/bs3compat-0.3.1/transition.js"></script><script src="libs/bs3compat-0.3.1/tabs.js"></script><script src="libs/bs3compat-0.3.1/bs3compat.js"></script><link href="libs/bs4_book-1.0.0/bs4_book.css" rel="stylesheet">
<script src="libs/bs4_book-1.0.0/bs4_book.js"></script><link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
<link rel="manifest" href="/site.webmanifest">
<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#d42d2d">
<meta name="apple-mobile-web-app-title" content="Data Science at the Command Line">
<meta name="application-name" content="Data Science at the Command Line">
<meta name="msapplication-TileColor" content="#b91d47">
<meta name="theme-color" content="#ffffff">
<script>
      (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
      (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
      m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
      })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
      ga('create', 'UA-43246574-3', 'auto');
      ga('send', 'pageview');
    </script><script src="https://cdnjs.cloudflare.com/ajax/libs/autocomplete.js/0.38.0/autocomplete.jquery.min.js" integrity="sha512-GU9ayf+66Xx2TmpxqJpliWbT5PiGYxpaG8rfnBEk1LL8l1KGkRShhngwdXK1UgqhAzWpZHSiYPc09/NwDQIGyg==" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mark.js/8.11.1/mark.min.js" integrity="sha512-5CYOlHXGh6QpOFA/TeTylKLWfB3ftPsde7AnmhuitiTX4K5SqCLBeKro6sPS8ilsz1Q4NRx3v8Ko2IBiszzdww==" crossorigin="anonymous"></script><!-- CSS --><link rel="stylesheet" href="dsatcl2e.css">
</head>
<body data-spy="scroll" data-target="#toc">

<div class="container-fluid">
<div class="row">
  <header class="col-sm-12 col-lg-2 sidebar sidebar-book"><a class="sr-only sr-only-focusable" href="#content">Skip to main content</a>

    <div class="d-flex align-items-start justify-content-between">
      <img id="cover" class="d-none d-lg-block" src="images/cover-small.png"><h1 class="d-lg-none">
        <a href="index.html" title="">Data Science at the Command Line, 2e</a>
      </h1>
      <button class="btn btn-outline-primary d-lg-none ml-2 mt-1" type="button" data-toggle="collapse" data-target="#main-nav" aria-expanded="true" aria-controls="main-nav"><i class="fas fa-bars"></i><span class="sr-only">Show table of contents</span></button>
    </div>

    <div id="main-nav" class="collapse-lg">
      <form role="search">
        <input id="search" class="form-control" type="search" placeholder="Search" aria-label="Search">
</form>
      <nav aria-label="Table of contents"><h2>Table of contents</h2>
        <ul class="book-toc list-unstyled">
<li><a class="" href="index.html">Welcome</a></li>
<li><a class="" href="foreword.html">Foreword</a></li>
<li><a class="" href="preface.html">Preface</a></li>
<li><a class="active" href="chapter-1-introduction.html"><span class="header-section-number">1</span> Introduction</a></li>
<li><a class="" href="chapter-2-getting-started.html"><span class="header-section-number">2</span> Getting Started</a></li>
<li><a class="" href="chapter-3-obtaining-data.html"><span class="header-section-number">3</span> Obtaining Data</a></li>
<li><a class="" href="chapter-4-creating-command-line-tools.html"><span class="header-section-number">4</span> Creating Command-line Tools</a></li>
<li><a class="" href="chapter-5-scrubbing-data.html"><span class="header-section-number">5</span> Scrubbing Data</a></li>
<li><a class="" href="chapter-6-project-management-with-make.html"><span class="header-section-number">6</span> Project Management with Make</a></li>
<li><a class="" href="chapter-7-exploring-data.html"><span class="header-section-number">7</span> Exploring Data</a></li>
<li><a class="" href="chapter-8-parallel-pipelines.html"><span class="header-section-number">8</span> Parallel Pipelines</a></li>
<li><a class="" href="chapter-9-modeling-data.html"><span class="header-section-number">9</span> Modeling Data</a></li>
<li><a class="" href="chapter-10-polyglot-data-science.html"><span class="header-section-number">10</span> Polyglot Data Science</a></li>
<li><a class="" href="chapter-11-conclusion.html"><span class="header-section-number">11</span> Conclusion</a></li>
<li><a class="" href="list-of-command-line-tools.html">List of Command-Line Tools</a></li>
</ul>

        <div class="book-extra">
          <p><a id="book-repo" href="https://github.com/jeroenjanssens/data-science-at-the-command-line">View book repository <i class=""></i></a></p>
        </div>

        <div>
          <a id="course-signup" href="/#course">Embrace the Command Line</a>
        </div>
      </nav>
</div>
  </header><main class="col-sm-12 col-md-9 col-lg-7" id="content"><div id="chapter-1-introduction" class="section level1" number="1">
<h1>
<span class="header-section-number">1</span> Introduction<a class="anchor" aria-label="anchor" href="#chapter-1-introduction"><i class="fas fa-link"></i></a>
</h1>
<p>This book is about doing data science at the command line.
My aim is to make you a more efficient and productive data scientist by teaching you how to leverage the power of the command line.</p>
<p>Having both the terms <em>data science</em> and <em>command line</em> in the title requires an explanation.
How can a technology that is over 50 years old<a class="footnote-ref" tabindex="0" data-toggle="popover" data-content='&lt;p&gt;The development of the UNIX operating system started back in 1969. It featured a command line since the beginning. The important concept of pipes, which I will discuss in &lt;a href="chapter-2-getting-started.html#essential-concepts"&gt;Section 2.3&lt;/a&gt;, was added in 1973.&lt;/p&gt;'><sup>1</sup></a> be of any use to a field that is only a few years young?</p>
<p>Today, data scientists can choose from an overwhelming collection of exciting technologies and programming languages.
Python, R, Julia, and Apache Spark are but a few examples.
You may already have experience in one or more of these.
If so, then why should you still care about the command line for doing data science?
What does the command line have to offer that these other technologies and programming languages do not?</p>
<p>These are all valid questions.
In this first chapter I will answer these questions as follows.
First, I provide a practical definition of data science that will act as the backbone of this book.
Second, I’ll list five important advantages of the command line.
By the end of this chapter I hope to have convinced you that the command line is indeed worth learning for doing data science.</p>
<div id="data-science-is-osemn" class="section level2" number="1.1">
<h2>
<span class="header-section-number">1.1</span> Data Science is OSEMN<a class="anchor" aria-label="anchor" href="#data-science-is-osemn"><i class="fas fa-link"></i></a>
</h2>
<p>The field of data science is still in its infancy, and as such, there exist various definitions of what it encompasses.
Throughout this book I employ a very practical definition by <span class="citation"><a href="#ref-Mason2010" role="doc-biblioref">Hilary Mason and Chris H. Wiggins</a><a class="footnote-ref" tabindex="0" data-toggle="popover" data-content='&lt;p&gt;&lt;a href="#ref-Mason2010" role="doc-biblioref"&gt;&lt;span&gt;“A Taxonomy of Data Science,”&lt;/span&gt; 2010, &lt;/a&gt;&lt;a href="http://www.dataists.com/2010/09/a-taxonomy-of-data-science" role="doc-biblioref"&gt;http://www.dataists.com/2010/09/a-taxonomy-of-data-science&lt;/a&gt;.&lt;/p&gt;'><sup>2</sup></a></span>.
They define data science according to the following five steps: (1) obtaining data, (2) scrubbing data, (3) exploring data, (4) modeling data, and (5) interpreting data.
Together, these steps form the OSEMN model (which is pronounced as <em>awesome</em>).
This definition serves as the backbone of this book because each step, (except step 5, interpreting data, which I explain below) has its own chapter.</p>
<p>Although the five steps are discussed in a linear and incremental fashion, in practice it is very common to move back and forth between them or to perform multiple steps at the same time.
Figure <a href="chapter-1-introduction.html#fig:diagram-osemn">1.1</a> illustrates that doing data science is an iterative and non-linear process.
For example, once you have modeled your data, and you look at the results, you may decide to go back to the scrubbing step to the adjust the features of the dataset.</p>
<div class="figure" style="text-align: center">
<span style="display:block;" id="fig:diagram-osemn"></span>
<img src="images/dscl_0101.png" alt="Doing data science is an iterative and non-linear process" width="90%"><p class="caption">
Figure 1.1: Doing data science is an iterative and non-linear process
</p>
</div>
<p>Below I explain what each step entails.</p>
<div id="obtaining-data" class="section level3" number="1.1.1">
<h3>
<span class="header-section-number">1.1.1</span> Obtaining Data<a class="anchor" aria-label="anchor" href="#obtaining-data"><i class="fas fa-link"></i></a>
</h3>
<p>Without any data, there is little data science you can do.
So the first step is obtaining data.
Unless you are fortunate enough to already possess data, you may need to do one or more of the following:</p>
<ul>
<li>Download data from another location (e.g., a webpage or server)</li>
<li>Query data from a database or API (e.g., MySQL or Twitter)</li>
<li>Extract data from another file (e.g., an HTML file or spreadsheet)</li>
<li>Generate data yourself (e.g., reading sensors or taking surveys)</li>
</ul>
<p>In <a href="chapter-3-obtaining-data.html#chapter-3-obtaining-data">Chapter 3</a>, I discuss several methods for obtaining data using the command line.
The obtained data will most likely be in either plain text, CSV, JSON, HTML, or XML format.
The next step is to scrub this data.</p>
</div>
<div id="scrubbing-data" class="section level3" number="1.1.2">
<h3>
<span class="header-section-number">1.1.2</span> Scrubbing Data<a class="anchor" aria-label="anchor" href="#scrubbing-data"><i class="fas fa-link"></i></a>
</h3>
<p>It is not uncommon that the obtained data has missing values, inconsistencies, errors, weird characters, or uninteresting columns.
In that case, you have to <em>scrub</em>, or clean, the data before you can do anything interesting with it.
Common scrubbing operations include:</p>
<ul>
<li>Filtering lines</li>
<li>Extracting certain columns</li>
<li>Replacing values</li>
<li>Extracting words</li>
<li>Handling missing values and duplicates</li>
<li>Converting data from one format to another</li>
</ul>
<p>While we data scientists love to create exciting data visualizations and insightful models (steps 3 and 4), usually much effort goes into obtaining and scrubbing the required data first (steps 1 and 2).
In <em>Data Jujitsu</em>, <span class="citation"><a href="#ref-Patil2012" role="doc-biblioref">DJ Patil</a><a class="footnote-ref" tabindex="0" data-toggle="popover" data-content='&lt;p&gt;&lt;a href="#ref-Patil2012" role="doc-biblioref"&gt;&lt;em&gt;Data Jujitsu&lt;/em&gt; (O’Reilly Media, 2012)&lt;/a&gt;.&lt;/p&gt;'><sup>3</sup></a></span> states that “80% of the work in any data project is in cleaning the data.”
In <a href="chapter-5-scrubbing-data.html#chapter-5-scrubbing-data">Chapter 5</a>, I demonstrate how the command line can help accomplish such data scrubbing operations.</p>
</div>
<div id="exploring-data" class="section level3" number="1.1.3">
<h3>
<span class="header-section-number">1.1.3</span> Exploring Data<a class="anchor" aria-label="anchor" href="#exploring-data"><i class="fas fa-link"></i></a>
</h3>
<p>Once you have scrubbed your data, you are ready to explore it.
This is where it gets interesting because when you’re exploring, you will truly get to know your data.
In <a href="chapter-7-exploring-data.html#chapter-7-exploring-data">Chapter 7</a> I show you how the command line can be used to:</p>
<ul>
<li>Look at your data</li>
<li>Derive statistics from your data</li>
<li>Create insightful visualizations</li>
</ul>
<p>Command-line tools introduced in <a href="chapter-7-exploring-data.html#chapter-7-exploring-data">Chapter 7</a> include: <code>csvstat</code><span class="citation"><a class="footnote-ref" tabindex="0" data-toggle="popover" data-content='&lt;p&gt;&lt;a href="#ref-csvstat" role="doc-biblioref"&gt;Christopher Groskopf, &lt;em&gt;&lt;span class="nocase"&gt;csvstat&lt;/span&gt; – Print Descriptive Statistics for Each Column in a &lt;span&gt;CSV&lt;/span&gt; File&lt;/em&gt;, version 1.0.5, 2020, &lt;/a&gt;&lt;a href="https://csvkit.rtfd.org" role="doc-biblioref"&gt;https://csvkit.rtfd.org&lt;/a&gt;.&lt;/p&gt;'><sup>4</sup></a></span> and <code>rush</code><span class="citation"><a class="footnote-ref" tabindex="0" data-toggle="popover" data-content='&lt;p&gt;&lt;a href="#ref-rush" role="doc-biblioref"&gt;Jeroen Janssens, &lt;em&gt;&lt;span class="nocase"&gt;rush&lt;/span&gt; – &lt;span&gt;R&lt;/span&gt; One-Liners from the Shell&lt;/em&gt;, version 0.1, 2021, &lt;/a&gt;&lt;a href="https://github.com/jeroenjanssens/rush" role="doc-biblioref"&gt;https://github.com/jeroenjanssens/rush&lt;/a&gt;.&lt;/p&gt;'><sup>5</sup></a></span>.</p>
</div>
<div id="modeling-data" class="section level3" number="1.1.4">
<h3>
<span class="header-section-number">1.1.4</span> Modeling Data<a class="anchor" aria-label="anchor" href="#modeling-data"><i class="fas fa-link"></i></a>
</h3>
<p>If you want to explain the data or predict what will happen, you probably want to create a statistical model of your data.
Techniques to create a model include clustering, classification, regression, and dimensionality reduction.
The command line is not suitable for programming a new type of model from scratch.
It is, however, very useful to be able to build a model from the command line.
In <a href="chapter-9-modeling-data.html#chapter-9-modeling-data">Chapter 9</a> I will introduce several command-line tools that either build a model locally or employ an API to perform the computation in the cloud.</p>
</div>
<div id="interpreting-data" class="section level3" number="1.1.5">
<h3>
<span class="header-section-number">1.1.5</span> Interpreting Data<a class="anchor" aria-label="anchor" href="#interpreting-data"><i class="fas fa-link"></i></a>
</h3>
<p>The final and perhaps most important step in the OSEMN model is interpreting data.
This step involves:</p>
<ul>
<li>Drawing conclusions from your data</li>
<li>Evaluating what your results mean</li>
<li>Communicating your result</li>
</ul>
<p>To be honest, the computer is of little use here, and the command line does not really come into play at this stage.
Once you have reached this step, it’s up to you.
This is the only step in the OSEMN model which does not have its own chapter.
Instead, I refer you to the book <em>Thinking with Data</em> by <span class="citation"><a href="#ref-Shron2014" role="doc-biblioref">Max Shron</a><a class="footnote-ref" tabindex="0" data-toggle="popover" data-content='&lt;p&gt;&lt;a href="#ref-Shron2014" role="doc-biblioref"&gt;&lt;em&gt;Thinking with Data&lt;/em&gt; (O’Reilly Media, 2014)&lt;/a&gt;.&lt;/p&gt;'><sup>6</sup></a></span>.</p>
</div>
</div>
<div id="intermezzo-chapters" class="section level2" number="1.2">
<h2>
<span class="header-section-number">1.2</span> Intermezzo Chapters<a class="anchor" aria-label="anchor" href="#intermezzo-chapters"><i class="fas fa-link"></i></a>
</h2>
<p>Besides the chapters that cover the OSEMN steps, there are four intermezzo chapters.
Each discusses a more general topic concerning data science, and how the command line is employed for that.
These topics are applicable to any step in the data science process.</p>
<p>In <a href="chapter-4-creating-command-line-tools.html#chapter-4-creating-command-line-tools">Chapter 4</a>, I discuss how to create reusable tools for the command line.
These personal tools can come from both long commands that you have typed on the command line, or from existing code that you have written in, say, Python or R.
Being able to create your own tools allows you to become more efficient and productive.</p>
<p>Because the command line is an interactive environment for doing data science, it can become challenging to keep track of your workflow.
In <a href="chapter-6-project-management-with-make.html#chapter-6-project-management-with-make">Chapter 6</a>, I demonstrate a command-line tool called <code>make</code>, which allows you to define your data science workflow in terms of tasks and the dependencies between them.
This tool increases the reproducibility of your workflow, not only for you but also for your colleagues and peers.</p>
<p>In <a href="chapter-8-parallel-pipelines.html#chapter-8-parallel-pipelines">Chapter 8</a>, I explain how your commands and tools can be sped up by running them in parallel.
Using a command-line tool called GNU Parallel<span class="citation"><a class="footnote-ref" tabindex="0" data-toggle="popover" data-content='&lt;p&gt;&lt;a href="#ref-parallel" role="doc-biblioref"&gt;Ole Tange, &lt;em&gt;&lt;span class="nocase"&gt;parallel&lt;/span&gt; – Build and Execute Shell Command Lines from Standard Input in Parallel&lt;/em&gt;, version 20161222, 2016, &lt;/a&gt;&lt;a href="https://www.gnu.org/software/parallel" role="doc-biblioref"&gt;https://www.gnu.org/software/parallel&lt;/a&gt;.&lt;/p&gt;'><sup>7</sup></a></span>, you can apply command-line tools to very large datasets and run them on multiple cores or even remote machines.</p>
<p>In <a href="chapter-10-polyglot-data-science.html#chapter-10-polyglot-data-science">Chapter 10</a>, I discuss how to employ the power of the command line in other environments and programming languages such as R, RStudio, Python, Jupyter Notebooks, and even Apache Spark.</p>
</div>
<div id="what-is-the-command-line" class="section level2" number="1.3">
<h2>
<span class="header-section-number">1.3</span> What is the Command Line?<a class="anchor" aria-label="anchor" href="#what-is-the-command-line"><i class="fas fa-link"></i></a>
</h2>
<p>Before I discuss <em>why</em> you should use the command line for data science, let’s take a peek at <em>what</em> the command line actually looks like (it may be already familiar to you).
Figure <a href="chapter-1-introduction.html#fig:mac-terminal">1.2</a> and Figure <a href="chapter-1-introduction.html#fig:ubuntu-terminal">1.3</a> show a screenshot of the command line as it appears by default on macOS and Ubuntu, respectively.
Ubuntu is a particular distribution of GNU/Linux, and it’s the one I’ll be using in this book.</p>
<div class="figure" style="text-align: center">
<span style="display:block;" id="fig:mac-terminal"></span>
<img src="images/screenshot_terminal_mac_dst.png" alt="Command line on macOS" width="90%"><p class="caption">
Figure 1.2: Command line on macOS
</p>
</div>
<div class="figure" style="text-align: center">
<span style="display:block;" id="fig:ubuntu-terminal"></span>
<img src="images/screenshot_terminal_ubuntu_dst.png" alt="Command line on Ubuntu" width="90%"><p class="caption">
Figure 1.3: Command line on Ubuntu
</p>
</div>
<p>The window shown in the two screenshots is called the <em>terminal</em>.
This is the program that enables you to interact with the shell.
It is the shell that executes the commands I type in.
In <a href="chapter-2-getting-started.html#chapter-2-getting-started">Chapter 2</a>, I explain these two terms in more detail.</p>

<div class="rmdnote">
I’m not showing the Microsoft Windows command line (also known as the Command Prompt or PowerShell), because it’s fundamentally different and incompatible with the commands presented in this book.
The good news is that you can install a Docker image on Microsoft Windows so that you’re able to follow along.
How to install the Docker image is explained in <a href="chapter-2-getting-started.html#chapter-2-getting-started">Chapter 2</a>.
</div>
<p>Typing commands is a very different way of interacting with your computer than through a <em>graphical user interface</em> (GUI).
If you are mostly used to processing data in, say, Microsoft Excel, then this approach may seem intimidating at first.
Don’t be afraid.
Trust me when I say that you’ll get used to working at the command line very quickly.</p>
<p>In this book, the commands that I type and the output that they generate are displayed as text.
For example, the contents of the terminal in the two screenshots would look like this:</p>
<pre><span style="font-weight: bold">$</span> <span style="color: #5f8700">whoami</span>
dst
 
<span style="font-weight: bold">$</span> <span style="color: #5f8700">date</span>
Tue Dec 14 11:43:30 AM CET 2021
 
<span style="font-weight: bold">$</span> <span style="color: #5f8700">echo</span> <span style="color: #af8700">'The command line is awesome!'</span> | <span style="color: #5f8700">cowsay</span> -f tux
 ______________________________
&lt; The command line is awesome! &gt;
 ------------------------------
   \
    \
        .--.
       |o_o |
       |:_/ |
      //   \ \
     (|     | )
    /'\_   _/`\
    \___)=(___/
 
 
<span style="font-weight: bold">$</span></pre>
<p>You’ll also notice that each command is preceded with a dollar sign (<strong><code>$</code></strong>).
This is called the prompt.
The prompt in the two screenshots showed more information, namely the username, the date, and a penguin.
It’s a convention to show only a dollar sign in examples, because the prompt (1) can change during a session (when you go to a different directory), (2) can be customized by the user (e.g., it can also show the time or the current <code>git</code><span class="citation"><a class="footnote-ref" tabindex="0" data-toggle="popover" data-content='&lt;p&gt;&lt;a href="#ref-git" role="doc-biblioref"&gt;Linus Torvalds and Junio C. Hamano, &lt;em&gt;&lt;span class="nocase"&gt;git&lt;/span&gt; – the Stupid Content Tracker&lt;/em&gt;, version 2.25.1, 2021, &lt;/a&gt;&lt;a href="https://git-scm.com" role="doc-biblioref"&gt;https://git-scm.com&lt;/a&gt;.&lt;/p&gt;'><sup>8</sup></a></span> branch you’re working on), and (3) is irrelevant for the commands themselves.</p>
<p>In the next chapter I’ll explain much more about essential command-line concepts.
Now it’s time to first explain <em>why</em> you should learn to use the command line for doing data science.</p>
</div>
<div id="why-data-science-at-the-command-line" class="section level2" number="1.4">
<h2>
<span class="header-section-number">1.4</span> Why Data Science at the Command Line?<a class="anchor" aria-label="anchor" href="#why-data-science-at-the-command-line"><i class="fas fa-link"></i></a>
</h2>
<p>The command line has many great advantages that can really make you a more efficient and productive data scientist.
Roughly grouping the advantages, the command line is: agile, augmenting, scalable, extensible, and ubiquitous.
I elaborate on each advantage below.</p>
<div id="the-command-line-is-agile" class="section level3" number="1.4.1">
<h3>
<span class="header-section-number">1.4.1</span> The Command Line is Agile<a class="anchor" aria-label="anchor" href="#the-command-line-is-agile"><i class="fas fa-link"></i></a>
</h3>
<p>The first advantage of the command line is that it allows you to be agile.
Data science has a very interactive and exploratory nature, and the environment that you work in needs to allow for that.
The command line achieves this by two means.</p>
<p>First, the command line provides a so-called <em>read-eval-print-loop</em> (REPL).
This means that you type in command, press <strong><code>Enter</code></strong>, and the command is evaluated immediately.
A REPL is often much more convenient for doing data science than the edit-compile-run-debug cycle associated with scripts, large programs, and, say, Hadoop jobs.
Your commands are executed immediately, may be stopped at will, and can be changed quickly.
This short iteration cycle really allows you to play with your data.</p>
<p>Second, the command line is very close to the file system.
Because data is the main ingredient for doing data science, it is important to be able to work easily with the files that contain your dataset.
The command line offers many convenient tools for this.</p>
</div>
<div id="the-command-line-is-augmenting" class="section level3" number="1.4.2">
<h3>
<span class="header-section-number">1.4.2</span> The Command Line is Augmenting<a class="anchor" aria-label="anchor" href="#the-command-line-is-augmenting"><i class="fas fa-link"></i></a>
</h3>
<p>The command line integrates well with other technologies.
Whatever technology your data science workflow currently includes (whether it’s R, Python, or Excel), please know that I’m not suggesting you abandon that workflow.
Instead, consider the command line as an augmenting technology that amplifies the technologies you’re currently employing.
It can do so in three ways.</p>
<p>First, the command line can act as a glue between many different data science tools.
One way to glue tools is by connecting the output from the first tool to the input of the second tool.
In <a href="chapter-2-getting-started.html#chapter-2-getting-started">Chapter 2</a> I explain how this works.</p>
<p>Second, you can often delegate tasks to the command line from your own environment.
For example, Python, R, and Apache Spark allow you to run command-line tools and capture their output.
I demonstrate this with examples in <a href="chapter-10-polyglot-data-science.html#chapter-10-polyglot-data-science">Chapter 10</a>.</p>
<p>Third, you can convert your code (e.g., a Python or R script) into a reusable command-line tool.
That way, it doesn’t matter anymore in what language it’s written.
Now, it can be used from the command line directly or from any environment that integrates with the command line as mentioned in the previous paragraph.
I explain how to this in <a href="chapter-4-creating-command-line-tools.html#chapter-4-creating-command-line-tools">Chapter 4</a>.</p>
<p>In the end, every technology has its strengths and weaknesses, so it’s good to know several and use whichever is most appropriate for the task at hand.
Sometimes that means using R, sometimes the command line, and sometimes even pen and paper.
By the end of this book you’ll have a solid understanding of when you should use the command line, and when you’re better off continuing with your favorite programming language or statistical computing environment.</p>
</div>
<div id="the-command-line-is-scalable" class="section level3" number="1.4.3">
<h3>
<span class="header-section-number">1.4.3</span> The Command Line is Scalable<a class="anchor" aria-label="anchor" href="#the-command-line-is-scalable"><i class="fas fa-link"></i></a>
</h3>
<p>As I’ve said before, working on the command line is very different from using a GUI.
On the command line you do things by typing, whereas with a GUI, you do things by pointing and clicking with a mouse.</p>
<p>Everything that you type manually on the command line can also be automated through scripts and tools.
This makes it very easy to re-run your commands in case you made a mistake, when the input data has changed, or because your colleague wants to perform the same analysis.
Moreover, your commands can be run at specific intervals, on a remote server, and in parallel on many chunks of data (more on that in <a href="chapter-8-parallel-pipelines.html#chapter-8-parallel-pipelines">Chapter 8</a>).</p>
<p>Because the command line is automatable, it becomes scalable and repeatable.
It’s not straightforward to automate pointing and clicking, which makes a GUI a less suitable environment for doing scalable and repeatable data science.</p>
</div>
<div id="the-command-line-is-extensible" class="section level3" number="1.4.4">
<h3>
<span class="header-section-number">1.4.4</span> The Command Line is Extensible<a class="anchor" aria-label="anchor" href="#the-command-line-is-extensible"><i class="fas fa-link"></i></a>
</h3>
<p>The command line itself was invented over 50 years ago.
Its core functionality has largely remained unchanged, but its <em>tools</em>, which are the workhorses of the command-line, are being developed on a daily basis.</p>
<p>The command line itself is language-agnostic.
This allows the command-line tools to be written in many different programming languages.
The open source community is producing many free and high-quality command-line tools that we can use for data science.</p>
<p>These command-line tools can work together, which makes the command line very flexible.
You can also create your own tools, allowing you to extending the effective functionality of the command line.</p>
</div>
<div id="the-command-line-is-ubiquitous" class="section level3" number="1.4.5">
<h3>
<span class="header-section-number">1.4.5</span> The Command Line is Ubiquitous<a class="anchor" aria-label="anchor" href="#the-command-line-is-ubiquitous"><i class="fas fa-link"></i></a>
</h3>
<p>Because the command line comes with any Unix-like operating system, including Ubuntu Linux and macOS, it can be found in many places.
Plus, 100% of the top 500 supercomputers are running Linux.<a class="footnote-ref" tabindex="0" data-toggle="popover" data-content='&lt;p&gt;See &lt;a href="https://top500.org/statistics/details/osfam/1/"&gt;TOP500&lt;/a&gt; which keeps track of how many super computers run Linux.&lt;/p&gt;'><sup>9</sup></a>
So, if you ever get your hands on one of those supercomputers (or if you ever find yourself in Jurassic Park with the doorlocks not working), you better know your way around the command line!</p>
<p>But Linux not only runs on supercomputers.
It also runs on servers, laptops, and embedded systems.
These days, many companies offer cloud computing, where you can easily launch new machines on the fly.
If you ever log in to such a machine (or a server in general), it’s almost certain that you’ll arrive at the command line.</p>
<p>It’s also important to note that the command line isn’t just a hype.
This technology has been around for more than five decades, and I’m convinced that it’s here to stay for another five.
Learning how to use the command line (for data science and in general) is therefore a worthwhile investment.</p>
</div>
</div>
<div id="summary" class="section level2" number="1.5">
<h2>
<span class="header-section-number">1.5</span> Summary<a class="anchor" aria-label="anchor" href="#summary"><i class="fas fa-link"></i></a>
</h2>
<p>In this chapter I have introduced you to OSEMN model for doing the data science, which I use as a guide throughout the book.
I have provided some background about the Unix command line and hopefully convinced you that it’s a suitable environment for doing data science.
In the next chapter I’m going to show you how to get started by installing the datasets and tools and explaining the fundamental concepts.</p>
</div>
<div id="for-further-exploration" class="section level2" number="1.6">
<h2>
<span class="header-section-number">1.6</span> For Further Exploration<a class="anchor" aria-label="anchor" href="#for-further-exploration"><i class="fas fa-link"></i></a>
</h2>
<ul>
<li>The book <em>UNIX: A History and a Memoir</em> by Brian W. Kernighan tells the story of Unix, explaining what it is, how it was developed, and why it matters.</li>
<li>In 2018 I gave a presentation titled <em>50 Reasons to Learn the Shell for Doing Data Science</em> at Strata London. You can <a href="https://datascienceatthecommandline.com/resources/50-reasons.pdf">read the slides</a> if you need even more convincing.</li>
<li>The short but sweet book <em>Thinking with Data</em> by Max Shron focuses on the <em>why</em> instead of the <em>how</em> and provides a framework for defining your data science project that will help you ask the right questions and solve the right problems.</li>
</ul>
</div>
</div>

  <div class="chapter-nav">
<div class="prev"><a href="preface.html">Preface</a></div>
<div class="next"><a href="chapter-2-getting-started.html"><span class="header-section-number">2</span> Getting Started</a></div>
</div></main><div class="col-md-3 col-lg-3 d-none d-md-block sidebar sidebar-chapter">
    <nav id="toc" data-toggle="toc" aria-label="On this page"><h2>On this page</h2>
      <ul class="nav navbar-nav">
<li><a class="nav-link" href="#chapter-1-introduction"><span class="header-section-number">1</span> Introduction</a></li>
<li>
<a class="nav-link" href="#data-science-is-osemn"><span class="header-section-number">1.1</span> Data Science is OSEMN</a><ul class="nav navbar-nav">
<li><a class="nav-link" href="#obtaining-data"><span class="header-section-number">1.1.1</span> Obtaining Data</a></li>
<li><a class="nav-link" href="#scrubbing-data"><span class="header-section-number">1.1.2</span> Scrubbing Data</a></li>
<li><a class="nav-link" href="#exploring-data"><span class="header-section-number">1.1.3</span> Exploring Data</a></li>
<li><a class="nav-link" href="#modeling-data"><span class="header-section-number">1.1.4</span> Modeling Data</a></li>
<li><a class="nav-link" href="#interpreting-data"><span class="header-section-number">1.1.5</span> Interpreting Data</a></li>
</ul>
</li>
<li><a class="nav-link" href="#intermezzo-chapters"><span class="header-section-number">1.2</span> Intermezzo Chapters</a></li>
<li><a class="nav-link" href="#what-is-the-command-line"><span class="header-section-number">1.3</span> What is the Command Line?</a></li>
<li>
<a class="nav-link" href="#why-data-science-at-the-command-line"><span class="header-section-number">1.4</span> Why Data Science at the Command Line?</a><ul class="nav navbar-nav">
<li><a class="nav-link" href="#the-command-line-is-agile"><span class="header-section-number">1.4.1</span> The Command Line is Agile</a></li>
<li><a class="nav-link" href="#the-command-line-is-augmenting"><span class="header-section-number">1.4.2</span> The Command Line is Augmenting</a></li>
<li><a class="nav-link" href="#the-command-line-is-scalable"><span class="header-section-number">1.4.3</span> The Command Line is Scalable</a></li>
<li><a class="nav-link" href="#the-command-line-is-extensible"><span class="header-section-number">1.4.4</span> The Command Line is Extensible</a></li>
<li><a class="nav-link" href="#the-command-line-is-ubiquitous"><span class="header-section-number">1.4.5</span> The Command Line is Ubiquitous</a></li>
</ul>
</li>
<li><a class="nav-link" href="#summary"><span class="header-section-number">1.5</span> Summary</a></li>
<li><a class="nav-link" href="#for-further-exploration"><span class="header-section-number">1.6</span> For Further Exploration</a></li>
</ul>

      <div class="book-extra">
        <ul class="list-unstyled">
<li><a id="book-source" href="https://github.com/jeroenjanssens/data-science-at-the-command-line/blob/master/book/2e/01.Rmd">View source <i class=""></i></a></li>
          <li><a id="book-edit" href="https://github.com/jeroenjanssens/data-science-at-the-command-line/edit/master/book/2e/01.Rmd">Edit this page <i class=""></i></a></li>
        </ul>
</div>
    </nav>
</div>

</div>
</div> <!-- .container -->

<footer class="bg-primary text-light mt-5"><div class="container-fluid">
    <div class="row">
      <div class="d-none d-lg-block col-lg-2 sidebar"></div>
      <div class="col-sm-12 col-md-9 col-lg-7 mt-3" style="max-width: 45rem;">
        <p><strong>Data Science at the Command Line, 2e</strong> by <a href="https://twitter.com/jeroenhjanssens" class="text-light">Jeroen Janssens</a>. Updated on December 14, 2021. This book was built by the <a class="text-light" href="https://bookdown.org">bookdown</a> R package.</p>
      </div>
      <div class="col-md-3 col-lg-3 d-none d-md-block sidebar"></div>
    </div>
  </div>
</footer>
</body>
</html>
